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Ibis Technical Keport was written to be one of a series of Eu.h 
reports that will be incorporated into a Computer Program Kanagenert 
Hanabook. As such, tiiis repart is directly applicable for use by 
Electrcmic Systems Division system program managers in the a^q.uisi- 
tion of computer program configuration items- 

Supplemental guidance concerning coaqniter program testing may be 
found in AFR 80-IU, l^st and Eyaluatitm of Systems, Subsystems, and 
Equipment . and AFSC Design Handbook IH k-2. Electronic Systems Test 
and Braluation . 

Since this report %ras written, Capt. Trainor has been reassigLei 
to the Air Force Avionics laboratory, Wri^t-Patterscn Air Force Base, 
Ojic, 

This technical report has beer reviewed and is approveo. 
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ABSTRACT 



This Technical Report addresses the geDeral area of coi^iuter prograa 
testing. Tbe test progrsB for eoBpoter prog r sM is, in ■any eases, 
dlffera:it froa tbat Tor equipaent due to tbe uniqueness z,T the cccQ^uter 
prograa ccafiguratlao Itea. ftcae, testii^ concepts for ccB^nter pro 
eraas ar« sot always videly understood. ISiis TecbnieciL Report attei^ts 
to clarify ccoputer progras testing requlreaoits and procedures by zo^- 
sidering such topics as test requireaests doeaperts and test plans/ 
procedures /reports. Tbe concepts of inforaal versus fozaal testing 
are introduced, vnicn lead to tbe subjects of prellnlnary and foraaJ. 
qualification testing. SubprogrsB, fanctiocal, and coapoter progras 
eosifignra^iaa itea levels otT testing are also explained, and these 
ideas are In turn related to infoival versus foraal testing. 
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SECTION I 

urnwDOCTiOR 

1. General. Testing perforaed a« an Integral p^^^t of the acquisition 
process is governed by AFF. 80-14 and is aodredsec in this docuaent as 
DevelopiKnt Test auad Evaluation (CT&E), and C-perational Test and Evalua- 
tion (0T4E). The purpose of -he total test erf ore is to verify the terfor- 
■ance requireaents and coa^liance vith specif icaz .ont of configuration 
item, subsysteas, and the total, integrated ajsz^a. 

2. Developsaent Test and Evaluation . Itoe DT4E effort is divided Into 
the two areEis of Cl/Suhsystem Test and System Test. Tne lunctlons of 
eacn of the two areas of lflt£ are given belov. 

a. Cl/Sutesystea Test . "Rie Cl/Subsystea nesting effcrt consists of 
the develcpc3>nt testing and evaluation of the individual configuratioD 
it^MS (CZs) . subsystems and, in certain ca«es, the coBplet ^ syst^. Tbe 
M.r ?oree actively particlps-tes in, evaluax-s, atd controls the Cl/Sub- 
systea testing; however, the tests are conducted prednwlnantly by the 
contractor vho is under Air Force Systeas Caannxd (AFSC) c ixectloo and 
control. 

(1) The overall objective of the Cl/Siobsysten e *f ort is the 
qualification of all CIs and subsysx^eas or segcents, thereby preparing 
each eleaent of the system for the subsequent Sy; "--=1 test jrograa. The 
total objective is fulfilled by the following subordinate <u:ca^llsh- 
Bents: 

(a) Siigloetrring test and evaluation necescary to the 
developoent of an acceptable design. 

(b) Preliminary Qualification Testing (PQ5) to confirm 
the functiorial integrity of mission critical functions. 

(c) Foraal ^caaliflcation Testing (PftT) of each CI, or 
group, to both environnent and functiooal performance requireaents. 

(d) Reliability teat and analysis which confiras reli- 
ability goa?s and defines potential probl€ 



(e) Integration of hardware, coaiputer prograas, and 
personnel subsystems. 

(f) Qualification of system segiaents or svbsysteas as 
specified Ir, perforaiance requireaents. 

(^) Govemment cu::;trol of the Cl/Subsystem test prograa is 
established primsxily througji provisions of the contract and reqoire- 
aentE of th- specifications. Although aovemaert personn 1 will 
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noraally oot c<»iduct any CZ/Sub8yet;em testing, it is intended that the 
SFO will designate govemoent representatives, on site, to perform as 
■onltors to ietemine the test progress, adherence to test procedures, 
validity of collected data, and perfomance of the equipoeat and coa- 
puter progriias. Normally, all Ccqiuter Program Conf Igurai ion Item. 
(CPCI) testing will be conducted as an Integral part of t} 3 Cl/Sub. 
system effort. 

b. System Test . The System testing effort consists of testing pt. 
evaluati<m spanning the integration of subsystems into a cooplete sys 
and developEnent tests of the coopleted system in as near an operational 
configuration and environiaent as practicable. System testiitg is an Air 
Force effort with ccxitractor participation, under AFSC directlcm and 
control, and with active operating and supporting coomand participation. 
Actual test operation and maintenance should be performed by military 
per^cnnel who have received formal systea tret?,ning. 

(. ) The objective of System testing is the fonal qualification 
of the syst'^ffl specification requirements. Specific objectives of the Sys- 
tem test effort sure: 

(a) Demonstrate that the system can perfoiji the mission 
as specified In Section 3 of the System Specification. 

(b) Verify results of Cl/Subsystem testing with Air Force 
crews in a live operational environment. 

(c) Qualify and/or demonstrate the perfornance of CIs 
which requl:-e full system operation for qualification. 

(d) Verify the adequacy and compatibility of the main- 
tenance and supply support concepts as developed. 

(e) Determine the SEifety characteristics > f the systec, 
and procedxires necessary to operate and mainteiin the system. 

(f ) Provide sufficiently trained personnel for the operat- 
ing COTEands to assvime operational evaluation tasks durln<T the OT&E phase . 

(g) Verify required technical handbooks aiid manuals. 

3- Operational Test and £-.-a.I'.iation . The CfTKFl testing ef.ort normally 
follows System testing and cccspletes the testing phase of the Program 
Management Plan (f*CP) . OTiE tests will oe conducted by t-.e ai>propriate 
operating comaand with technical support by Air Force Sys .ems Coonand 
and Air Force Logistics Command. 



SECTION II 
TEST DOCUMERTATIOK 



1. Introduction . There arc several types of te-r. documei nation «rJ.cb, 
collectively, fora tne ba^ls for an effective test program. Broadly, 
these can "oe classed as : 

a. Test Requiresaer.ts, which are Section 'j of eaci: System i.Type k) 
or CI (Type 3 or C) Specification. 

b. TcBi; Flans, which are usually tee product of a validation i^iase 
and should reflect the overall planning for cest ajid evaluation of the 
system or a subsystem. 

c. Test Procedures, which are the detailed procelur? i information 
for conducting each test delineated in the tes* p^ans (re: . "b' above). 

d. Tes" Reports, which suaanarize the results and analysic of each 
test conduc-ed. 

2. Test Rei . iireaer.ts Docuae'its . Section h, 'Quality Assva-ance,' of each 
System (Typ*. A) or CI (Type B or C) Specification containt specified con- 
tractual re iiireuients for testing of the respective syster or CI. This 
specificati n section should depict h requirement to test each performance 
and design . equireaent contained i.. Sectiori " of tne spec ficatton. 
Generally, Section U of a Type A specification will speci: y requlreaents 
for System Test, and Section U of a Type B or C speciflca Ion will 
specify requirements for Cl/Subsystem Tept. 
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a. Section ^, S ys '-er. Specification . S.'Ction 
contains the requireaentc for the bystem test prcgrrja. Tsiese require- 
ments oust be relatable to t>erfonaanc3/desl;5n requirements stated in 
Section 3. Tnerefore, Section •* will nomsfwlly be liaiited to system 
level test equiremcnts, out will alao lnclud<» requirements for Cl/Sub- 
system engireering teste, qualifications, and reliability tests which 
can be acco:.s)ll3red only at the system test location(8) . The principal 
content of Section U. .however, Is the specif Ication of Sy t«m test 
requirements . 
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Test reau rements are 



developed initially oy tr.e contractor for incorporition l.ito Section 4 
(Quality Assurance) of the Fart I specif ication. Tnis section should 
Identify test laethodB (to the level of detail nf^cessary to clearly 
estabxlsh the scope and ^cc^u•acy of the aeth^as) to be employed In 
qualifying the C?CI a^alns all performance -^.^ oesigi, requlreaents 
specified Vj Section 3 oC tha,t CI specificati a.. In addition. It should 
Identlf;^ nqulrements for governaent-fumishc i equipment and facilities 
to support the contractor's computer program test a.id evaluation, as well 



as designate those perfonnance characteristics to be demonstrb .ed aDd/or 
verified during prelialnary qualification tests and demonstrations, 
foraal qualification testing, and Systea testing. 

(l) Requireaentb of Section k should be specified to ae level 
of detail vhldi: 

(a) Designates verification requirements and methods for 
each performance/ design requirenent identified In Section 3 • The Liethci 
of verification to be specified may Inclule inspection, review of analyt- 
ical data, denonstration tests, and review of te. t data. 



(b) Clearly establishes the scope and accuracy of the test 



method. 



(2) Types of Cl/Subsyst«m tests which inay be spec fled iii 
Section U of a CPCI Specification are: 

(a) Con^Juter Prograoning Test and Evaluation which are 
tests conducted prinarlly to support the design and develcpnent process, 
■niey are listed in the specificatiCHi only when they meet one of the 
following criteria: 

1, They are Intended to be the only soc.-ce of data to 
qualify spec:.flc requirements in Section 3- 

2. They oust be accooipllshed as part of an integrated 
test prograa Involving other systeas/equiiMBent/programs . 

^. TSjey require the use of government -1 imished test 
facilities or equipment. 

(b) Preliminary Qualification Tests, which txe toraal teste 
oriented primarily towards verifying portions of the CPCI prior to formaJ. 
qualificatloo tests of the Integrated CPCI. 

(c) PonvO. Qualificaticm Tests, which are formal tests 
oriented primarily towards testing of the Integrated CPCI, using oper- 
ationally configured equipment. 

(3) The Systen test reqiiirements to be identified in Section U 
concern those performance/design requlreoents which canno* be verified 
ontil System testing. Es^haais should be placed on minimizing these 
types of requirements and, if possltle, eliminating them ^together, 
"nils will ensure that the CPCI that is Cl/Subeystem testeu has in fact 
attained a hlg^ degree of confidence. Also, this will allow the System 
test program to proceed with its prime objective, testing of system 
level requirements. 

(k) An iaqportant point to remember is the differentiation 
between Section k and other testing documents. Section '' , by virtue of 
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belBg part of the Part I CPCI Specif Icatloo, Is a contractual docinent, 
and the contractor is required to conduct only those tests that are 
called out in that section. Any changes or deviations from these re- 
quireaents (once baselined) aust he approved by the Systeik Progran 
Office (SPO) through Engineering Change ProposauL (ECP) actlcas. Thus, 
careful attention nust he given to ensuring that the testing has be?n 
scoped properly, and the test requlreme: tb are complete a.id acceptable. 

3. Test Plans/Procedures/Heports . Figure 1 shows n. typi ^al test docu- 
nentation tree for a large system. The tree !£• ccnstrxictea from the 
list of approved tesi plan/procedure/report data item descriptions (DIDs) 
contained in AFSCM/AFLCM 310-1, Vol II. 

a. Syst e m Test Flan . The System Test Flan, DID T- 101-1, is the 
"top docuaent' for the test p3*ograsi which structures and JLuifies all 
subsequent test plans. Its purpose is to provide an overall outline 
of the total system test program to include plsjining factrs, objectives, 
and scope of all phases of the test program. The system -contractor pr** 
pares the System Test Flan (DID T-lOl-l), normally for de ivery in the 
C(mcept\2al or early Validation Phase. This plan will con ain basic test 
planning information for all phases of the test program a-.d cover the 
life cycle of the system through the end of the Pull Seal' Developnent 
Phase. The scope and level of detail of the infomatlcm xs sufficiently 
broad and c:^mprehensive to provide basic test inputs to t e Program 
Management "Ian (PMF) and sufficiently detailed to provid,- the basis 
for prepari.ig all subsequent test plans — particularly Cl/'^ubsystem 
(DID T-102-i and DTD T- 103-1) and Syston (DID T- 106-2) plans. 

(1) The System Test Plan (DID T-lOl-l) is based .ipon the test 
concepts (uid requirements contained In the Program Managei-ient Plan (FMP) , 
the System Specification, and related system engineering ocumentatlcm. 
It covers all aspects of the system, including hardware, ::; acllltles , 
computer programs, personnel, and procedural data with re-pect to such 
consideration aa the following: 

(a) Organizational responsibilities for testing. 

(b) Basic system test concepts and objectives for Cl/Sub- 
system. System, Implementation, and Acceptance tests. 



(c) Overall system test operations, including test control 
requirements and test support reqxilrements . 

(d) Test evaluation requlz*ements . 

(e) Test reporting requirements 

(f) Overall test schedules. 
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(2) A primary function of the System 'Pest Plem (DID T-lOl-l) 
Is to guide the contractor's planning, analysis, and system engineer- 
ing activities related to the test areas. During tbe Validation Phase 
the contractors expand basic planning and guidance InforBatlon contained 
in the System Test Pl«ui (DID T-lOl-l) and, in effect, replace, it with 
the following docunents: 

(a) Cl/Subsystem Test Plan (DID T- 102-1 and T-103-1}. 

(b) Inputs to the System Test Flan (DID T-10&-2). 

(c) Inputs to the liplenentation Test Plan (imiltlslte 
systems; DID T^llU-l) . 

(3) Special problems to be considered concem.lng cr tgjuter pro- 
grams may iuclude provisions for conqputing equipi&ent required for con- 
tractor development and testing of cooiputer programs. The Cl-Subs/stea 
test cycle for computer program CIs will nonaally begin early during 
the Full Scale Development Phase; it may have to oe inltl&tcid tislng a 
prototype coi^puter emd peripheral equipment, or in some cases by sIibu 
lation on existing conputers. As development of the CPCI progresses, 
testing will encoopcMs progressively expanded groups of r:^tlnes, and 
will require additional Items of conputlng equipment for reaillstlc test- 
ing of functions. Preliadnary qu&lificatioa tests/ demonstration will 
nozmally occur at the contractor's plant, or other locations of available 
equipment. Formal qiialification of the cooiputer program CI may require 
scheduled time at the System test site for testing those computer pro- 
gram functions that depend upon the operationally configured systeny' 
eqiilpiaent and for which simulation is not adequate. For a multisite 
system which requires adaptation of tbe operational ccniputer program 

CI at each site location, the expected sequence of events at each fol- 
low-on site will be installation and checkout of equlpiaent at the site 
facility, installation testing, adaptation, installation and checkout 
of computer programs, and ijgqplementatlon testing (See Figure 2). 

b. Cl/Subsystem Teat Flaa/Procedures/Reports (Equipment) . The Cl/ 
Subsystem Test Plan/Procedures data Item, DID T- 102-1, is the overall 
planning document for the Cl/Subsystem test program. It is subordinate 
(in the test planning tree) only to the System Test Plan (DID T-lOl-l). 
Tbls docuflient should provide ccaqplete planning information on each Cl/ 
Subsystem test specified in each of the CI speclflcatioos on contract — 
i.e.. System Specification, CI Specification, Critical Component Speci- 
fication, Military Specification, and other contractual docvuaents that 
Include Cl/Subsystem test methods and success criteria. Rote that test 
planning and procedures infonpation for Cl/Subsystem testing of computer 
programs should not be Included in this document. In tbe case where the 
system involves cooqputer programs, an additional test data item is used 
to detail the planning and procedural information for testing of the 
coomiter programs. Ibis document is Cl/Subsystem Test Plan/Procedures 
(Computer Program), DID T-103-1, which should be referenced in DID T-102-] 
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Tbi0 data Itea is noraally obtained In the Validation Fhase as a cob- 
plete plan and individual test infoxaatlon sheets are then updated in 
the Full Scale Development -gbaae prior to each test. Test reports are 
obtained under DID T-lld-1. 

c. Cl/Subsystem Test Flans/ProeedureB/Report (Conputer Programs). 
The coaputer progran Cl/Subsysten Test Plan/Procedures, DID T-103-I, is 
a subordinate and supplementary docuaent to the overall Cl/Subsyst^ 
Test Plan/Procedure, DID T-102-1. When the system involves computer 
programs and eqiiipoents, DID T-103-I should be placed on contract 
along with DID T- 102-1; DID T-102-1 will apply to the Cl/Subsystem test- 
ing of all equlpoents and reference DID T-103-1 which will cover the CI/ 
Subsystem testing for the cooQwter programs. 

(1) The Cl/Subsystem Test Plan for cootputer programs Is a 
contractor-prepared document which establishes criteria, ><eneral methods, 
responsibilities, and overall planning for Cl/Subsysteo testing of a 
CFCI. normally, che test planning inforasitlon is obtained in the Vali- 
dation Riase as a cosrplete plan applicable to all coiqputer programs of 
the system. Generally, only one plan (and one volume) will be prepared 
for a single system or contract — this allows for the presentation of an 
Integrated test plan which will apply to all coQiuter programs for a 
particular system. Exception to this guidance can be obtained for the 
Individual system. Sections ot the plan which apply to counter pro- 
grsBS for subsystems for which Inadequate Information exists at time of 
writing of *he plan, can be updated at a later date. Wheri this Is the 
case, the section should be Included nonetheless with the remark, "to 
be completed later." 

(a) The plan should contain detailed lnfonn4,tloo concern- • 
Ing the Ifliplementatlon of preliminary and formal qualification tests, 
along such lines as the following: 

1. Locations at which the tests will be conducted, 
and schedules relative to mllest<mes In the overall acqulslti<» schedule. 

2. General methods for preparation of Input data— 
i.e., simulation and/or generation vehicles to be used. 

^. General procedures for test conduct, and respon- 
sibilities for test direction, operation, and observation. 

4. General procedures for analysis of test results. 

3- Requirements for other caaputer programs, equip- 
ment, and facilities. 

6. Personnel requirements, including nuaibers, re- 
sp<»)slbllltles, and particular knowledge and skills required. 
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(b) The plan aay also set forth the req»iireir«iit8 and 
procedures for controlling emd docuaeotlng the Cl/Siibsystea test 
prograa, including procedures for preparing, reviewing and revising 
docuaentation of specific test procedures; requii:*e9Dents and pro- 
ce<iure8 for preparing and reviewing reports of individual <i\AJ>i_L^"'.- 
catloD tests; suonarles of the Cl/Subsystem test program or phages 
thereof; and other reports related to the Cl/Subsystem test activity. 

(2) The Cl/Subsystem Test Procedures are produced ty thu con- 
tractor during the Full Scale Developaent Fbase. 

(a) For each indivlducO. CI/SubsyBteo Qualificatiim Test 
(PQT or FQT) for a CPCI, the contractor will prepare a Cl/Subsystein 
Test Procedure in accordance with Section 2 of DID T-103-I. These 
procedures are prepared increaentally during the Pull Scale Developaent 
Phase and sutsiltted prior to the test date of the PQT or FQT to which 
they apply. Inforaation Included in a test procedure includes the 
follo»dJig: 

1. Locaticm and schedule of the test, oriefings, de- 
briefings, and any associated data reduct-lc»/ analysis . 

2. References to applicable test plan, specifications, 
manuals and handbooks. 

^. Detailed objectives of the test. 

U. Requirements and responsibilities for console 
operators, -est directors, technical consul tamts, data analysts, cr 
other essential test personnel. 

2- Requlreaents for other caoQiuter prograas (other 
than the CPCI being tested) or equipment. 

6. Test operating procedures to specify how to 
initiate the coBQniter program operation, maintain the conputer program 
operation, and terminate and/or restart the coiqwter program pperation. 

2* A dsxialled test description tor each test (or 
portion of a test) to be perfonaed. "Riis description should Include 
detailed Information on test Inputs, outputs, events, expected res\;lts, 
reactions to be verified, and methods of verification. 

8. Requirements and procedures for recording, re- 
duction and analysis of test data. 

(3) The Cl/Subsystem Test Repoirts for caoQniter program tests 
aure obtained under DID T-II8-I. These are contractor prepared docu- 
ments, coopiled incrementally during the FUll Scale Developnent Riase. 
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Generally, one test report (in accordance with DID T-llB-1) Is prepared 
for each PQT or FQT conducted. 

d. Syst<« Test Plan/Procedure/Report . DID T-106-2 Is the basic 
docuaent \iti.ch pronrides the overall integrated outline of the Systea 
Test Progrea. It includes all planning factors, scope, detailed test 
objectives, identificaticn of test areas, rasponslbllitles of partici- 
pating agercles, and associated inforaati<xi necessary to lapleaent the 
BinljauB acceptable perforaance requireaents of tJoe Prograa Nanageaent 
Directive (H<D) and/ or toe System Specif icaticm. NorTaallv, the con- 
tractor will prepare tbe Systea Test Flan and Procedures ;dID T-106-2) 
with the intent being to establish a test and evaluation prograa to 
ensure that the systea/equipnent/cooputer programs seet the "^^ *»<■■'- 
acceptable performance requireaents of the t)G> and Systea/CI Specifi- 
cations in as realistic and coaplete an operational envircnaent as 
practicable . Test Reports are obtained under DID T- 120-2 . 

e. Installation Testing . Installation Testir^, normally conducted 
after coo^letion of each installation, ixK:ludes ?reshaked>wn Tests, 
vhich are p-erfonoed as a cao&inatlon allgnaent check and ""^st to assure 
that the installation is properly coopleted; Shakedown Tests, which are 
perfomed to assure that all detected narginal parts and :fiterial have 
been eliminated and that the installaticni is ready for op -rational 
tests; and "oe rations* T?st8, which are perforael to deaonstrate that 
the equipaen-L is properly nstalled and Is capable of perfOTBiag its 
operational olssioc up ''-.o a specified inxerface with other portions of 
the subsystea/ systea. 

(1) The Installation and Checkout Flan (I4C), DIT T-112-1, is 
initially -prepared by tbe oontx'actor during the Validaticn Phase and 
later expar led to reflect the system engineering and detail design 
activity at the Pull Scale Developsaeat Phase. Procedures are prepared 
to iopleaent the plan, beised vpon assenibly levels of coaqpjoents and 
consideration of all interfau^es. In general, individual iteas are in- 
stalled, physically and functionally checked out, then B9 -ed to other 
subsystens. This graduate'', process is applied successively until tbe 
eaiq>lete systea is ready for systea tests. 

(2) IBC plans are reviewed and approved by tbe SPO and the 
procedures selectively reviewed and approved. 

(3' Adaptatl-in, i:is^allation, and checkout (AI8C) of tbe ct:«- 
puter program will ;ioraEall-/ occur after installatlcHi and checkout of 
equipaent and facilities. Normally, the work will be accoaplished by 
the contractor Ir; accToance with the approved plnns and procedures 
obtained under Din T-ii.?-^. At the Systea Test Site, AI«C of the com- 
puter progr on may be followed by Formal Qualification Testing prior to 
initiation of the System Test. Based upon the experience ^aixked at the 
System Tes*. Site, the rliC plan may oe modified and updated for use at 
each follow -on site. AI^ of coaputer programs at follow-on sites will 
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include the saae basic activities as for AliC at the Systea t^'St site 
and will occur in the saae general sequence, i.e., followiotf I&C of the 
equlpaent/facilltles, but prior to the start of Japleaents' ioi: Tests 
(see Figure 2). For coiqjuter prograas, appendices na^ be used to c?v«r 
iinique adaptation feaiiires for each site. 

(a) The coiqiuter prograa contractor aust Identi-'y require- 
■ents for AI&C of the cca^Niter prograa at the S/stoa test site, and for 
aoltislte s/steas, aust also include requir^aectr for AI8C at ^ubseqjent 
site Installations. Docuaentaxion sh-ulc Incluce a derailed description 
of all aspects of the SLdaptAtlcm, Installatlcn and chscjco^t activities. 
Detailed scedules should be provided and all support reqilreaents 
clearly lder.tif led. Including required coBputer time, reqolreasits for 
other systea equlpaents such as coHainlcation, display ca^^'^les, etc., 
and requlreioents for trained personnel. Detailed inforaa.ioa should 
also be provided concemlng the training of personnel for the field 
locations, scheduled aoveaent of field personnel, local t ansportaticD, 
office space and facilities, and living quarters at remot-ly located 
sites. Procedures should be prepared Indicating in detail the exact 
Banner in which the plan vl3J. be lapleaented . 



NC^: "Adaptation" refers to the process of inserting In' 
prograa the coded data which are apprcpiriate to the geogr 
cbaracteris ics of the given site. "Installation' refers 
of the code:- instruction/data Into u^.e cooSHitine equlpaen 
with equipment installation, coogputer program installatio 
Involves relatively Insignificant effort or tli»e. The aa. 
Aiac is the checkout activity, which aay in seme cases In 
of new eleaients, coaplllng, and extensive debugging, as w 
nel training and prellalnary rehearsals of FQT procedures 



o the coaputer 
ipby or other 
to Insertlcm 
; In contrast 
typically 
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olve asseably 
11 as person- 



f . Ii^leaentatloc Testing . Bq>leaentatlon Testing is a concept which 
has been derined to aeet the specific requirenents of Bult,lslte (i.e., 
ground installation, airborne vehicles, space platforas, or tactical 
eaplaceaent.':} electronic systeas. Tbe concept is based ca experience 
which has shown that additional testing beymd Installatl^ai Testing say 
be required at successive ciperating sites to insure that these sites, 
individually and collectively, function as paits of a sysoea In accor- 
dance with the requlreaents of the systea perforaance specification. 
These tests encoapass real-tlae functiooal testing perf or wd at the 
systea level with the purpose of exposing faults and providing a denioo- 
stration to the user that the installed site Is ready foz cqieratiooal 
use. 



(l! The relation of the Systea Testing and Lqjleaentatioo Test- 
ing should be one of decreasing cooplexity and sophistication. To the 
extent tha'. systea pez^ormance requlreaents set forth in the systea 
speclficat: 3n have been deaonstrated by Systea testing, a complete re- 
petltltxi o:' the effort for each follow-on systoa is not warranted. 
Rather, spi siflc functional perforaance aeasures which acequately define 
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BjmXtm resiKffise ar« Identified at •& •«;'ly point in tlae, and the actual 
raluea for such perfoaraance aeasures (figures of aerit) are deteraloed 
during Systea testing. Siese figures of aerit are used dur^ug Ivole- 
aentatloc Testing as the criteria toy ^Aiich to gauge fUnctl ^&I aoequacy 
of the systea. In general, I^leaent&tioc Tesx . should not attea5)t to 
dvq>llcate the eoiQirehensive testing accoaQ>li8hed during Systea testing. 

(2) To insure operatiooal validity, the user as well a^ the 
systec designer aost participate in :^leaejtatioc Test platJilng. "Sie 
user is interested in a test that vlU uzfover tectjii^al inc ^^>atibilities 
or functioiaal and operatiocal deficiencies. Thus, tLe :Lazner La vhlch 
the systea is to be eaployed is a vital factor in test planning. 

(3) Tbe lBQ>le9ectatior: Test Plan and Procedures arc tcrtrac tor- 
prepared docuaents obtained under DID T-11^-1. The lopleaectatloc Test 
Flan defines the overall scope of the iJ5>le-2entatior. tc-st.-, the objiec- 
tives, aethsds, and siqiport re^uireaents for -Jie coz^oi^ct .f this phaae 
of testing. The laspleBenxatiOQ Test Procecures (one se\ ?f p. ocediures 
for each test conducted) outline a s^ep-by-step sez of instructioiis and 
criteria for each test specified in the piaji. Test resul'-b are reported 
under DID T-119-1. 



(4) The coaputer prograa contractor ( s ) prepare i puts to the 
Ii^leaentatiOQ Test Plan/Procedures aaicb the same as for .he S/st^ 
Test Plan/ Procedures. Siiice iapleseatation tes^ir.^ is at ti:.e systea 
level, oQ^hasis vill be placed on those aspects of the cctiputer prograa 
perforaaoce which depend u?cn co^lex Interjections with other CrCIs, 
personnel, rquipaent, and data links during operation of the entire 
systea. 

(a) Special eophasis will be given tc those portions of 
the CPCIs (generally the operatiooal CPCI) wh.ich vary froa site-to-slte, 
i.e., envirooaental data values, special adaptatiori paraaeters, unlque- 
to-slte interfaces, and site-peculiar progrea routines. 

(b) Inputs tc the la^leaentation Test Plan relative to 
cas^Miter prograas should be siallar to the inputs to the SysT^ea Test 
Plan (DO T- 106-2) . In general, they should esQscasize those scpects of 
systen perforaaoce which are closely associated with or dependent upoo 
perfozaance of the CPCI(8). Frequently, the inputs aay represent a 
selected subset of the Systea Test Plan (DID T-lOS-2) inputs. 

k. Section 4, CPCI Fart IT Specification . Koraally, Section H cf a CI 
Part H Specification will coataln quality assurance provisions and 
acceptance test requireaents for follow-(»*. production iteas. However, 
unlike equipaenx iteas, there is no follov-on production process for a 
coa^^uter prograa itea; thus, the concept cf acceptance tests fcr pro- 
duction iteas will not apply to a CPCI. Instead., Sectiou J* of a CFCI 
Part II Specification should contain two subsections: 
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a. Test Plan/Procedure Cross Reference Index . This subsection 
should cGoxaln a cross-reference dla^raa depicting each fxmccion (as 
delineated in the CPCI Fart I Specif Icattcm) and relating thesi func- 
tloea to the correspcmding test plan/procedures that were used to 
qualify the IndiTldual requireaeits of the CPCI. 

b. Other Quality Assurance PrsvlsioDS . Tnis subsection ■ .oiild 
reference aad/or specify tee test/verification recuirejaects , aezhoca , 
and procedures which apply to preparation and duplication ot the zcsl- 
puter prograa (i.e., tapes, card decks, etc.)* 
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COMPOTER PROGRAM CI/SUBSYSTSM TEETHC 
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1. Introduction . Tiie Cl/Subsy?te»= testing of CPCIs serves a four'cid 
purpose : 

a. To e6t«iMish each CPC.l as a qualified end Item suitable for entry 
into the System Test Projrrsa. This qualification is accooplisbed by 
verifying the performance and design requlr-anenzs of zhe CPCI Part I 
Specification. 

b. To furnish the procurla? a<?ency vlth the proper visibility re- 
quired for effective Eanagement of th* systea- This is accon?)ll8hed 
through Judicious schedulirj; cf ?5Ts , thus establishing aiilestocea which 
vlll provide inslj^.t into the progress of the 3?CI design and devalop- 
aeot. If problems are encountered in the cociputer prograa area, early 
detection can be aade ar.d more effe-tive sumageaent and engineertrjg cao 
be focused on these potential problem areas. 



c. To serve as a standard or "straw nari about 
can develop his internal verification procedures. 



vhich vhe contractor 



d. To develop a corsprenensive test (called FQT) vhlch can be utilised 
after FCA as an -ingcir^ ti^st tool/procedure fr-r retes^ing the CPCI. POT 
is not only used for fcraal verification prior to JTA but will be con- 
tinuously updated ar.d ueod tc rctsct t^f» CPCI wh°riev€r rhan^^es are taadf 
to the progran. 

2. Types of Cl/Subsygten CFCI Testing . Cl/Snbsystem teeming of CPCIe 
can be broadly divided into t-jo types — inforaial testing and formal test- 
ing. The basic diff'^r-'nce oetve-?;; therp *>T-e^ r,*--°rjb f -to the documen- 
tation requirsaents ; IrSoiiz&X t^^stln^^ Tr-i'iiz?r .,he cc-cractvir's ioteraal 
test dr>cuBent«-tir:-, r.~r2\.r'>j' ar.c pro-'^oji:- «??•; ''crrm''t t«>'?tlag is ccHsducted 
in accordance vlth Air Fcrre approved test plans arid procedures . 
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CI/'?-ihr:--t,'-a ir.f~-nal '^"tlrg 'referred to as 

'. r:.r.i ^/'\Lu'iM,"n . ~:'"*-E w-iil usually form the bulX 
Zl/cMczyptcrB. test '•ffort. It is designed to be the 



Cc?TDut.^r Proi^r^in "": 

of the contractor's 

cootractC'>-'s "In- bouse' testing and requires no government appi*or.-cd test 

plan's /Tirore iurcH . '"•ener-' lly , the entir'? I.-form.l t«r*. effort ^1X1 be 

docunent-ed ir .trrr.p.^ >- b> -r:? -rrtrnr'-.o-, ro"* s'-'^h 'rf~-aRtloc will be 

tmde available t- fi" prr.-arlng ajcenc - -nly rn oe-ar 

begins when thr flr^t n'.:pr.rD«-:-a?n la c-.%-l'-.'' rrd ortir 

Pull Scale Development Phase - 



Informal testing 
- -.hrriughout the 



(l) Inforpial testing :;-. rr'-.ai-^'j. rr r.hrr'«» Ic^lr f.->r a «t^y?-by- 
otcp validation of the CPCI; 
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(a) Suoprograa level (paxameter tests) 

(b) Functioiial level (assenibly tests) 

(c) CPCI level (assenbly tests) 

(2) Farameter tests are tests of the irifonnation processing 
logic of the inrilviduaJ. CrCa. Sets of inputs are prepare i which cau£«e 
all the logic paths to be exercised. The resultaat outputs are checkea 
against hand calculations ^ich show whether the CPC has been codec 
correctly . 

(3) Assembly w^its are tests of groups of CFCs w.;ich perfont 
a f\mcti(» of the CPCI, or in later stages, iaany or ail o: the func- 
tions of the CPCI. Interfaces between the CPCs as well as overall 
information processing are tested. Simulated inputs to t e CPCI are 
\i8ed, and the CPCI is operated in real-time or siniulated eal-tlme 
mode. 

(U) The informal test activities are conducted solely for the 
purpose of providing information required for the developmental process . 
As such, they do not require recognit^lon by the procuring agency and 
would not Dtmally be identified in the Cl/Subsystea Test Plan. The 
planning ana conduct of infonnal testing is carried out ir. accordance 
with the contractor's internal managemenr procedures, and within the 
tiae coD8tr!*ints iagxjso- oy formally sche<-uled reviews ac -. qualification 
testing. Each CPC or iubprogram must pass, through a seri.s of stages 
and iterations, consisting of such operations as desk che :klng^ elimi- 
nation of illegal instruct lo.'is, parameter tests, function 1I. testing with 
ccmtrolled data inputs, usaembly with additional coiQ>oner .s of the CPCI, 
asseaibly testing, perfor7.anc6: testing with sibilated inpv js and, finally, 
performance testing in the {system under conditions of live operations. 

(5) Tne Cl/Svosys^-em Test Plan will not generally include plan- 
ning infom-ition for contractor testing which is conducted as an integral 
part of the design ana development process. However, the plan will 
typically oetall the contractor's requirements for goverx^aent facilities 
and other Bi^port requirea for conducting the tests. I^ese will include 
requirements for cooputing and peripheral equipment which must be fur- 
nished or otherwise maae availab...e for the ccmtractor's v^se on an ap- 
propriate schedule during ::he Tull Scale Development Phase. Where 
formal qualification of tne CPCI is scheduled to occur at the System 
test site. It will normally be preceded by a phase of contractor test 
and evaluation associateu with adaptation, installation, and checkout 
of the CPCI(s) at the sif: The Cl/Subsystem Test Plan i.hould include 
(or reference other plarjvxng documents ^ich include) requirements for 
scheduled jse of the facility, system equipment, other cooputer program 
CIs, per8o:j3el, and other items needed to support the coodaet of these 
contractor tests. 
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b. Formal Testing . Cl/Subeystem foraial testing Is •nbvt portion of 
Cl/Subsyatea testing vdilcb is conducted In accordance wito Air Force 
approved test plans/procedures. TkaxB, the Air Force has explicit con- 
trol of tbe type, nuniber, and frequency of tests to be pe rf omed. 

(1) The Cl/Subsystes fonaal test effort is divided jitc two 
distinct types of testing— PrellBlnarj' Qualification Testing (PWl) and 
Poraal Qualification Testing (PQT) . PQT is designed to be an incre- 
mental process which will provide the procxirlng agency proper visibility 
and control of the coBpuxer program developaent during the luae period 
between the Critical Design Review (CIS?) aod PQT. FVi is designed to be 
a complete and con^jrehensive test of the CPCI In a "one step" faahioc 
Ju«t prior to PCA. 

(a) Preliminary Qualification Testing (?QT) le coi^p>OBed 
of function level tests (assembly tests) which are condiic ed In accor- 
dance with Air Force approved test plans/procedures procu.-4d under 
DID T- 103-1. PQT is to be an incremental process whlcn occurs bevwifer 
CDR and FQT of a CPCI's developaent process. For each function (of the 
Fart, I CPCI Specification) which is designated for testing during fXJJT, 
a separate test procedure I3 written and a formal test conducted. Motfe 
that, generally, not all functions of a CPCI are tested -iurlr^g Fqtr since 
experience nas proven that It is both too costly and too tlBe-cons'jaing • 
Insteeui, onxy designated fxuictions of tbe CPCI are tested during PQT; 
thus, the problem becomes one of the selection of those fi;iictl0!n8 which 
should undergo PQT. Selection of these functions should ; e baaed on 
the following: 

1. A PQT should be condacted for e'.^b Turci-ion that 
is "time-critical" to the oevelopaent of the CPCI, subsyscem, or syeten. 
For exai^le, the coapiler f-jnction of a utility CPCI may be designated 
'time-critical" if the developaent of tbe remaining ccoputer programs 
depend on the timely development of a ccs^iler to be used for ccaftillDg 
the other programs. Tbe development o.f the executive function of an 
operational CPCI may eLLso be designated ' tlue-critlci\l ' since the 
orderly progression from parameter testing co assembly tasting of the 
operational CPCI would require the timely developaent of the executive 
fimetion . 

2. A PicJT shculd be conducted for each functlcm ttiat 
is "performance-crltlcflO. to the development of the CPCI, subaysteiD, or 
system. For exaa^jle, the executive function coulC be designated "per- 
formance-critical" as well as "time-critical due to tbe utilization of 
various exotic scheduling techniques. As another exasqile, the tracking 
function of a CPCI for a,-; or.-line radar system snight be designated "j«r- 
formance-critlceLL' due \^ je utilitation of i^ew tracking, smoothing, 
and filtering techniques . 

(b) Prelimlrjhjry Qusdif Icati en 'I'eBts (PQTs) shouJ 1 nori«ally 
be conducted at a contractor's development facility (i.e., contractor's 
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plant or other location of available equipment), typlcall:' using ccn- 
tpolled inputs specifically prepared for the test/demonBtration purpose. 
The Test Plan should outline the sequence of individual and/or assembled 
CPC tests, identify the CPCI perforawice/iieslgn requireaents to be veri- 
fied at each PQT, and identify special sljBulatioi./recordlng equipnent 
or other suprport requireiaents for the PQT program. Each Test Procedures 
doeuaent shcold be completed by the contractor and submitted to the 
monitoring agency sufficiently in advance of the scheduleU test session 
(i.e., two to four weeks) to pemit review and analysis f the procedures 
prior to witnessing the treating operatiCMss . The f ora&t nd content of a 
typical Test Procedures doeuaent are specified in DID T- 103-1. 

(c) Poi^aal Qualification Testing (PQT), unlike Infonsal 
testing and PQT, Is not an Increaental process. FQT is designed to be 
an integrated and coaQxrebensive functional test of the CPCI as a whole, 
and ususdly is conducted in one continuous tliae period Juot prior to 
FCA. Each function of the CPCI (as delineated in rhe CPCI Part I Spec5.- 
flcatlon) is tested during FQ7, regardless of the amount of Infoxval 
testing and PQT c<mducted previously. Since ?QT is conducted during 
the on-goln? design process, the PQT test procedures becoue obsolete 
within a fe. weeks after the PQT tests. In contrast, since F(JT is con- 
ducted after the design process cxilmlnates (just prior to PCA), the FQfT 
test procedures can be maintained end \Q>dated and used th:'-oughout the 
reaalnder of the FtQl Scale Developi&ent and Deployment Fh-ises. Thus, 
each time a change is iaiplenented in a CPCI, the FQT pr'' -^dures can be 
used as a test tool to retest the CPCI and ensiure that it still functions 
in accordance with the fuccti(»al requirements set forth In the Part I 
Specification. 

1. For ~he less coB^lex CPCIs, or for those (i.e., 
utility) which are~relat,I .ely insensitive to the system Oi^erations, 
formal qualification will usually be conducted at the contractor's 
development facility. However, for a large operational CrCI, the sheer 
coB^lexity of the performance requirements may dictate that FQT cai^ only 
be conducted in the context of tV^e operationally-configured system. In- 
cluding personnel and conmunications . Hence, for these cases, PQJE may 
often not to conducted at the contractor's plant (development f«i'?lllty), 
but may req\iire uae of the System test site prior to the l>eglnnlng of 
System testing. 

2. In t-e event that a cocnslex operatlonaa CPCI is 
scheduled for formal qu3Lll.'icstion at the System test site, scheduling 
of the FOX will normalxy be prior to the start of formal System test- 
ing, but following a period of contractor adaptation, instsllaticn, and 
creckout of the CPCI ir. '^te previously Installed and tested operational 
equipoent/faciiities . 

3- In either case, PftT will involve the use of sim- 
ulated and controlli.-vi Inruta which can be designed to cover the expected 
ranges of "ariables, syc-em operational modes and conditions. Including 
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capacities and Halts. Pull qualification of certain fuiictions nay 
depend upon subsequent System testing with live inputs and cossBunl- 
cations . 

U. The procedures for Fomal Qualifies acn Tests 
(ref DID T-IO3-I) will be la the sane format p-s the PQT procedures 
prepared during design and de/eloiaaent. However, these procedures will 
enphasiz^ verifying that all functional requiremejas are met, baaed on 
the demonstrated performance using the coniplete CFCI. Since the PQ;T 
procedures require approval by the nonitorlng agency, they should be 
submitted initially in prelijainary form. Review and iiialysis may resvilt 
in proposed changes which aust be resolved and reflected in revised 
preparations for the testing activity. Hence, the preliminary docunent 
should be cropleted in sufficient tine (i.e., three months prior to 
scheduled KjT) for taic potentially lengthy process to be accoinplished. 

3. Levels of Cl/Subsystem CFCI Testing . As mentioned eaiiier, testing 
of a CPCI is required at throe levels: subprogram level, functional 
level, a:id CPCI level. Regardless of the type of testing to be per- 
formed (i.e., informal versus formal testing), these three levels apply 
equally to each. This relationship is portrayed in Figure 3. 

a. Subprogram Testing . Subprogram testing, sometimes called para- 
meter testing, is directed toward ensuring that each subprogram (CPC or 
leaser entity) interprets its inputs correctly, successfully performs 
all tasks defined In the subprogram: coding specifications, and adheres 
to prescribe! coding conventions and standards. Subprogram testing is 
the most detiiled and basic testing that is performed upon coaputer 
programs. Since concentration is on testing basic, easily managed units 
of code, the detection, isolation and correction of errors that are 
exceedingly dlfficvLLt to detect, isolate and correct in lax«r phases of 
testi.^g is greatly facilitated, nierefore, detailed attention must be 
given to the specification of rigid test requirements so that errors 
are not overlooked that will later cause vmnecessary testing and analysis. 
Sach subprogram should be specifically tested for arithmetic and logical 
accuracy and limitations, usually without ..c quiring cooBunicatlon with 
related subprograms or external equipment. Subprogram testing ccnoMnces 
follo%d.ng thf' develoiaBent of a set of coded instructions that have been 
subjected to detailed visual verification (code checking) by the respon- 
sible prograaiaers , and automatic code analysis by the applicable compiler 
and/or aasexbler to eliminate errors in keypunching, formatting of code, 
etc. In addition, a thorough technical review of the subprogram should 
be conducted by contractor personnel to ensure that violations of cod- 
ing conventions and rtandards have been corrected prior to on-coaQniter 
testing. The subprr re . is then operated with a range of data that 
forces use of ail decision points and processing paths. The test re- 
sults are analyzed to determine whether the derived results are con- 
sistent with the input data, the results expected from the selected 
input parameters, and tne operation of the subprogram as defined In cod- 
ing specifications. 
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b. Pupctlonal Area Testing . Fuxwtional area testing, sagwtlmes 
rei'erred to as string or asseaibly testing, is directed to^ ard ensuring 
that selected sets of functionally related siibprograas interpret all 
inputs correctly, successfully perform all processing taslis specified 
In performance specifications, end generate output data that satisfies 
the input requirenents of other interfacing functional areas or equip- 
Bients. Functional area testing is an extension of subprogram testing 
with en^basjs l^ing pleused on inter- program coinounicationB and pro> 
cesslng of data vithin a defined grouping of subprograms or coiQtonents. 
Subprogram testing, on the other hand, concentrates en intra- program 
coaminications and data processing. Examples of functional areas are 
radar inputs processing and correlation, data link inputs processing, 
and weapons guidance. Each functional area is composed of one or more 
subprograms that must accept as inputs ot^ier functioneUL area outputs, 
or inputs received via the operating hardware, and proces? tbea in 
accordance with functional perfoznance specifications. It must also 
prepare data for use by other sets of functionally relate a subprograms 
as specified in detailed coding specifications. In aone cases, (l>e., 
tape read), functional area testing may be the first phase of testizig 
performed, nils will most often occur when subprogram testing requires 
the excessively expensive simulation of interfacing hardware. 

(1) Functional area testing is usually the first point at vdiich 
Interactloce with a control subprogram are examined. Ide"J.ly, the ex- 
ecutive or control function is tne first to enter the functional area 
test phase, since tr^e other functions can be more conveniently tested 
with it. In actual practice, however, the control functl .'n may not be 
available or far enough along in development for this purpose. In this 
event, other methods for control may be utilized. AlthoiL^^ many func- 
tional areas are usually being tested at the same time, tne scope of 
testing gradually Increases through higher and higher levels of asseaibly 
as functlOTd become verified until the computer program is ready to 
undergo CPCI testing. In practice then, functional area testing begins 
as "high-l«-vel" sxibprograa testing and phases out as "low-level" CPCI 
testing. 

(2) Throughout this phase of testing, the perforiaance of the 
computer program is checked against that specified in performance re- 
quirements decisions and limits that result from an inteiface between 
subprograms, nierefore, conditions that are of consequence to only one 
subprogram, such as proper exits in subroutine decision paths, are 
better tested in the subprogram test phase when the problems of deflBlng 
and establishing the proper test sitiiations are not so complex. In ad- 
dition to differences in levels of concentration, functional area test- 
ing examines interactions be ./een all functionally related components 
^lle CPCI testing concentrates on interactions between functions. Also, 
Internally stored inputs tl ^t are provided to the siibprograms during 

the functional area test paase usually result from the insertion of test 
data by a *^«8t input tool. CPCI testing, on the other hfond, uses CI 
level inputs that are provided either by other CPCIs, or by system in- 
put data generators. 
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c. CFCI Testliag . CPCI testing Is directed toward ensuring that the 
total package of cooQuter prqgraa coBQ)oneat8 that ecm^rise the CPCI 
correctly Interprets all li^put aessage typeo and values, acc^ttng 
those that are legea and properly disposing at all others; properly 
processes all Internally stored data; and correctly foraats, arranges, 
and outputs all required systen data. Ihus, CPCI testing verifies that 
the CPCs operating as the CFCI fulfill the requlreaenta of perforaance 
specifications. Whereas subprogran testing concentrates oa verifying 
the correct operation of Individual conputer prograa CQaq>onent8, and 
functional area testing emphasizes inter-program casnunlcatlons vlth).n 
separable functional areas, CPCI testing concentrates on verifying that 
the caq>lete set of ccnputer prograa functional areas correctly int^r- 
flicts vltb each other. Therefore, CPCI testing must be aecam>ll8hed 
after functional ckrea testing and prior to, or in parallel with, test- 
ing with other CIs. CPCI testing can be conducted at either the ccm- 
tractor's In-plant test facility or an installation facility (either 
operatl<xially eiq>loyed or a nonoperational test site). During the 
initial stages of CPCI testing, the bulk of the testing 1^ conducted 
in-house using simulated iiiputs. This provides a hl^ degree of con- 
fidtnee prior to the release of a CI to a site location. During the 
later stages of CPCI testing, eau^taaala shifts from the testing of func- 
tiooal coaaninlcaticxt to verification of overall CFCI perforaance in an 
operat local (or very nearly so) environment. 
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